Principal Component Analysis (PCA) 
PCA (Principal Component Analysis) is a dimensionality reduction technique and helps us to reduce the number of features in a dataset while keeping the most important information. It changes complex datasets by transforming correlated features into a smaller set of uncorrelated components. 
How Principal Component Analysis Works
PCA uses linear algebra to transform data into new features called principal components. It finds these by calculating eigenvectors (directions) and eigenvalues (importance) from the covariance matrix. PCA selects the top components with the highest eigenvalues and projects the data onto them simplify the dataset.
Step 1: Standardize the Data
Different features may have different units and scales like salary vs. age. To compare them fairly PCA first standardizes the data by making each feature have:
A mean of 0
A standard deviation of 1
Step 2: Calculate Covariance Matrix
Next PCA calculates the covariance matrix to see how features relate to each other whether they increase or decrease together. The covariance between two features x1 and x2 ​is:
step 3: Find the Principal Components
PCA identifies new axes where the data spreads out the most:
1st Principal Component (PC1): The direction of maximum variance (most spread).
2nd Principal Component (PC2): The next best direction, perpendicular to PC1 and so on.
These directions come from the eigenvectors of the covariance matrix and their importance is measured by eigenvalues. For a square matrix A an eigenvector X (a non-zero vector) and its corresponding eigenvalue λ satisfy:
Step 4: Pick the Top Directions & Transform Data
After calculating the eigenvalues and eigenvectors PCA ranks them by the amount of information they capture. We then:
Select the top k components that capture most of the variance like 95%.
Transform the original dataset by projecting it onto these top components.
This means we reduce the number of features (dimensions) while keeping the important patterns in the data.
In the above image the original dataset has two features "Radius" and "Area" represented by the black axes. PCA identifies two new directions: PC₁ and PC₂ which are the principal components.
These new axes are rotated versions of the original ones. PC₁ captures the maximum variance in the data meaning it holds the most information while PC₂ captures the remaining variance and is perpendicular to PC₁.
The spread of data is much wider along PC₁ than along PC₂. This is why PC₁ is chosen for dimensionality reduction. By projecting the data points (blue crosses) onto PC₁ we effectively transform the 2D data into 1D and retain most of the important structure and patterns.